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Introduction: Educational Accountability Models in 
Transition 



State educational accountability models are in transition. Whether modifying the 
present accountability system to comply with existing state and federal requirements or 
anticipating new ones— such as the U.S. Department of Education's (ED) Race to the Top 
competition — recording the experiences of state education agencies (SEAs) that are 
currently undergoing transitions is both informative and important. Despite varied 
contexts, demands, and priorities, states charged with implementing transitions in their 
accountability models may find the experiences of the Accountability Systems and 
Reporting (ASR) collaborative member states useful in their own planning. 

Defining accountability has 
become more complex as our 
understanding of it has grown. In 
the past, definitions have 
focused primarily on the 
interaction of goals, indicators, 
decision rules, and 

consequences. Although those components are still central to any accountability model, 
more recently the focus has expanded to include building capacity and providing 
appropriate supports. The state experiences described herein reflect the changing 
purpose of accountability from identifying and punishing ineffective schools and districts 
to providing appropriate supports and cultivating effectiveness. 

In 2007 the ASR collaborative commissioned a paper titled Key Elements for Educational 
Accountability Models (Perie, Park, & Klau 2007). The paper was the culmination of 
discussions and analysis conducted by state members and consultants concerning the 
theory, research, and practice of educational accountability. The authors identified 
seven components they believe must be considered in developing or modifying an 
accountability system: goals, performance indicators, design decisions, consequences, 
communication, support, and system evaluation, monitoring, and improvement. 1 Given 
the dynamic nature of accountability in many states, the advent of a new federal 
education administration, and the prospect of a coming reauthorization of the 
Elementary and Secondary Education Act (ESEA), a follow-up paper on several states' 
experiences undergoing transitions is both timely and worthwhile. 

The audience for this paper is educational leaders responsible for the development, 
implementation, and evaluation of large-scale, school- and district-based state 
accountability systems. 



The purpose of accountability is not 
simply to identify and punish ineffective 
schools and districts, but to provide 
appropriate supports to cultivate 
effectiveness. 







1 For a broader discussion of these components, please refer to (Perie, Park, & Klau 2007) Key Elements 
for Educational Accountability Models, available online at 
www.ccsso.org/publications/details.cfm?PublicationlD=359. 
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Methodology 

We asked representatives from 10 ASR collaborative member states to contribute their 
insights with respect to accountability transitions that were implemented in the last few 
years or are planned in future years. 

Using the Key Elements paper as a starting point for identifying possible topics, we asked 
state education leaders from participating states to share their experiences of an 
accountability transition in their state. Each member was asked the following: 

1. State event producing transition: What was the accountability transition in 
your state? 

2. Context of transition: What triggered this transition? What was the event or 
policy decision? 

3. Effects of transition: What components of the state accountability system 
were or will be affected by the transition? 

4. Lessons learned: What lessons were learned from the transition in your 
state? 

5. Changes in goals: How have the goals of your state accountability system 
changed due to this transition? 

6. Communication, training, and support: What were or will be your plans for 
communication, training, and support? 

7. Evaluation and system monitoring: What were or will be your plans for 
evaluation and system monitoring? 

ASR project consultants and staff collated and edited the responses, which were then 
provided to the initiating SEA leader as well as a second SEA leader for validation 
purposes. Contributing states were then given the opportunity to review the final text 
prior to publication. 

Please note that the information contained herein does not necessarily provide a 
comprehensive picture of a state's experience with transitions; details were selected 
based on responses from ASR members. 
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How to Use This Guide 



To help the reader locate the information that is most useful to them, the content is 
organized in two ways: 

• Components of accountability: Readers wishing to understand how ASR states 
have dealt with transition within a particular accountability component (e.g., 
goals of accountability) can read just those sections. 

. Individual state case studies: Readers interested in the context underlying a 
given state's transition— particularly if a certain component above resonates 
with them— will find this section useful. 

Components of Accountability 



Goals: What are the purposes, uses, and contexts for the 

ACCOUNTABILITY SYSTEM? 

Goals refer to the purposes, uses, and contexts for the accountability system. We 
distinguish between the purposes, which provide an overarching reason for using an 
accountability system, and the goals, which specify the intended outcomes. The key 
activity is to develop an "explicit theory of action" linking intended outcomes to the 
various indicators and supports provided. 

• Alabama is implementing the National Governor's Association (NGA) cohort 
graduation rate (pages 10-11). 

• Hawaii is developing new codes to account for transfer students in four-year 
graduation rate calculations (pages 11-12). 

• Iowa is improving the accuracy of cohort graduation rate data (pages 12-14). 

• Kansas is aligning its high school end-of-course tests to successful course 
completion (pages 14-15). 

• Kentucky is responding to a legislative push to develop a new system of 
standards and assessments, coupled with the desire to minimize the time spent 
by teachers and students on the state assessment (pages 15-17). 

• Massachusetts is incorporating the four-year cohort graduation rate as a 
component of Adequate Yearly Progress (AYP) determinations (pages 17-21). 

• Michigan is developing a work skills assessment and college entrance 
examinations (pages 21-24). 

. Minnesota is implementing a "second generation of high school assessments" 
(pages 24-25). 
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Performance Indicators: What indicators will be used to 

MEASURE PERFORMANCE AND IMPROVEMENT? 

A major issue in any accountability system is the question of what to measure- 
performance indicators. One must examine the data that are available, the targets of 
the data collection, and the timing of the data collection. Consideration also needs to be 
given to ensuring the reliability and validity of the data. 

. Alabama is implementing a new data collection process (pages 10-11). 

. Kansas is increasing flexibility and accuracy of high school course completion 
(pages 14-15). 

• Hawaii is reporting adjusted graduation rates alongside standard cohort 
graduation rates so that local educators can map the progress and attainment of 
students continuously enrolled in their schools (pages 11-12). 

• Iowa's expanded data collection system allows expanded analyses at the point 
when students enter and exit the public education system (pages 12-14). 

. Michigan's schools had to transition from a three-week testing window at the 
high school level to giving the test to all students on the same day for each of 
three days (pages 21-24). 

• Minnesota is aligning passing grades in high school courses to No Child Left 
Behind (NCLB) proficiency levels (pages 24-25). 

. Nebraska is shifting from a locally based state assessment system to a common 
system statewide, in part to ensure more valid comparisons among districts 
(pages 25-27). 

. West Virginia is employing multiple indicators of student performance to create 
a holistic picture of student performance (pages 28-31). 

Design Decisions: How will indicators be used to make 

DECISIONS ABOUT TEACHER, SCHOOL, AND DISTRICT EFFECTIVENESS? 

Once policymakers have decided on a set of indicators, the next question is how to use 
them to make decisions about teacher, school, and district effectiveness— the design 
decisions. This issue gets at one of the main points of discussion about the ESEA 
regulations— whether, for example, school effectiveness is best measured using a 
status, improvement, or growth model— or some combination of these. Policymakers 
face design decisions such as how to combine indicators to make decisions about 
students, teachers, schools, and districts. For example, will the indicators be combined 
in a compensatory fashion, where low performance on one measure can be offset by 
high performance on another? Or will there be a minimum level of performance set for 
each measure? In addition, decisions must be made regarding school classification, such 
as how high to set a bar, how often to raise a bar, and how to balance reliability and 
validity concerns. In all cases, the decisions are guided by the goals of the system. 
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• Kansas found the need to develop consistent policies, procedures, and business 
rules governing when students would be eligible to take retests in subjects 
covered under its Opportunity to Learn (OTL) program (pages 14-15). 

• Massachusetts recognized the need to report graduation rates in a timely way, 
but that objective had to be balanced with ensuring that the reported rates are 
as accurate as possible (pages 17-21). 

• West Virginia is tying its accountability index to school accreditation (pages 28- 
31). 

Consequences: What rewards or sanctions will be tied to the 

ACCOUNTABILITY SYSTEM? 

Policymakers implement consequences tied to the goals of the accountability system. In 
most accountability models, schools that meet the goals are rewarded, and schools that 
fail to meet the goals are sanctioned and receive some type of intervention or support. 
States must determine appropriate consequences, target them to the appropriate 
people and organizations, apply them effectively, and monitor their impact on student 
achievement and other outcomes. 

• Massachusetts is incorporating accountability for student subgroups in 
graduation rate calculations (pages 17-21). 

Communication: How will data be provided to stakeholders and the 

GENERAL PUBLIC IN A MANNER THAT IS BOTH UNDERSTANDABLE AND USEFUL? 

Communication includes communication about the goals and consequences of the 
accountability system as well as the communication of results, such as score reporting. 
This element focuses on providing data to stakeholders and the general public in a 
manner that is both understandable and useful. 

• Alabama has placed a priority on communicating details about its transition to 
the NGA cohort graduation rate via multiple, yet cost-effective means, including 
"living documents" and webcasts (pages 10-11). 

• Hawaii is using two separate graduation rates over a two-year transition period 
(pages 11-12). 

• Iowa is implementing a coding process beginning in 2008, which will affect AYP 
in 2010 (pages 12-14). 

• Kansas is utilizing a two-year cycle to communicate its new approach to 
assessment to stakeholders (pages 14-15). 

. Kentucky is working collaboratively with state legislators to craft the final 
language defining the state's accountability system (pages 15-17). 
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• Massachusetts convened stakeholders from across the state to address 
graduation rates with respect to AYP determinations and build school/district 
capacity in increasing graduation rates (pages 17-21). 

• Minnesota is phasing in one high school assessment per year (pages 24-25). 

• Nebraska is leveraging the expertise of local educators in developing its 
statewide assessment system amid rapid changes in legislation (pages 25-27). 

• West Virginia is working on changes to standards and assessments as part of a 
five-year process (pages 28-31). 

Support: What resources and services will support schools and 

DISTRICTS AS THEY TRY TO ATTAIN THE GOALS OF THE ACCOUNTABILITY 
SYSTEM? 

Support focuses on resources and services that support schools and districts as they try 
to attain the goals of the accountability system. The focus is on the roles of state, 
district, and school agents in developing a plan for school improvement, communicating 
this plan, and providing the necessary resources to ensure that each school can meet 
the overarching goals. 

• Alabama has placed a priority on using multiple, yet cost-effective, training and 
support mechanisms, including "living documents" and webcasts, to convey 
information about its transition to the NGA cohort graduation rate (pages 10- 
11 ). 

. Iowa utilizes the state's fiber optic network, the Iowa Communications Network, 
as well as a series of larger, all-inclusive, face-to-face meetings on data topics 
(pages 12-14). 

• Massachusetts piloted an early warning system to help local educators identify 
and intervene with students at risk of not graduating on time (pages 17-21). 

. West Virginia provides extensive training to educators on standards and 
assessments (pages 28-31). 

System Evaluation, Monitoring, and Improvement: What are the 

MECHANISMS FOR CONTINUALLY ANALYZING AND ADJUSTING THE MODEL TO 
ENSURE THAT THE GOALS ARE MET? 

System evaluation, monitoring, and improvement focuses on the mechanisms for 
continually analyzing and adjusting the accountability system appropriately. Successful 
systems develop an evaluation plan and use the results of the evaluation to make 
improvements. This evaluation should also answer questions regarding the effectiveness 
of various rewards and sanctions as well as other intervention or support strategies. 

• Alabama incorporates an opportunity for district review in conjunction with 
quality checks (pages 10-11). 
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• Hawaii convened an adjusted graduation cohort workgroup (pages 11-12). 

• Iowa utilizes a three-step data collection process: training, testing, and 
production (pages 12-14). 

. Kentucky works with special advisory groups that represent those involved, 
including the state board of education, superintendents advisory, DAC advisory 
and other key groups (pages 15-17). 

• Nebraska leverages a long-standing evaluation contract with the University of 
Nebraska as well as many external experts (pages 25-27). 



State Case Studies 

Alabama's Transition to the NCLB/National Governor's 
Association Methodology for Calculating Cohort Graduation 
Rates 

In 2007 Alabama adopted the 
National Governor's Association 
(NGA) methodology for calculating 
cohort graduation rates, with the 
goal of reporting the new rate 
beginning with the 2012 
graduating cohort. It replaced the 
National Center for Education 
Statistics (NCES) leaver rate as the 
additional AYP indicator for high 
schools. 

The NGA rate, as recalculated 
under this transition, will be lower 
than previous rates, which is being 
communicated to stakeholders in 
advance. 

Alabama has learned a number of 
lessons that can be shared with 
other states undergoing similar 
transitions: 

• Transfers versus dropouts: 

Because of the 
accountability implications 
associated with low 
graduation rates, the 
accurate reporting of 



Communication and support 

Alabama has invested considerable effort in 
communicating details of this transition both 
within the different departments of the state 
education agency and externally through 
training sessions with local school districts, 
accountability coordinators, principals, 
counselors, and other parties. 

Training has focused specifically on ensuring 
a clear understanding of the transition details 
and timeline, with special attention given to 
the new data documentation required. 

Communication devices include the posting of 
a "living document" on the state’s website, to 
which changes are made and communicated 
to stakeholders on a regular basis. The 
website also hosts a variety of supporting 
materials, including presentations and a 
transition timeline. Costs for communication, 
training, and support have been intentionally 
kept low due to economic constraints at the 
state and local levels. As such, Alabama uses 
webcasts as the primary communication tool. 
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students who transfer out of high school versus those who drop out has been a 
matter for concern. This concern has been mitigated by Alabama's recent 
legislative act requiring exit interviews with any students wishing to leave school 
prior to graduation. An additional byproduct of these exit interviews, which are 
designed to encourage students to stay in school, is that a school is able to 
accurately determine if the student is simply transferring or if the student is 
dropping out. 

• Missing records: Missing records in data collection resulted in the need to use 
unofficial replicated data from the local systems in order to accurately track 
student movement through the four years of high school. 

. Midyear promotion and first-time status: The majority of students begin high 
school in ninth grade; however, the system required business rules to handle the 
tracking of students promoted at midyear. 

• Mobility tracking: Alabama discovered that tracking students as they enter and 
leave schools and districts is a difficult and complex process. Solutions include 
the use of a unique student identifier for all students in the state, performing 
sufficient data quality checks, and ensuring that local school districts have the 
opportunity to review the data and make corrections prior to the public release. 

. Communication and support: In addition to the need to communicate the 
difference between the NCES and NGA methodologies— and their impact on 
accountability decisions— Alabama has found that professional development to 
all stakeholders is essential, especially with regard to accurate data reporting at 
the district level. At the state level, Alabama has learned the importance of 
involving all offices in the transition process. 

At the time of this writing, Alabama is considering plans for monitoring and evaluating 
the system, such as an interactive online portal that would allow stakeholders with 
access to student data the opportunity to track and verify the status of individual 
students as they move through high school, as well as perform calculations. 

Hawaii's Incorporation of New Codes to Include "Transfers-In" 
into Graduation Rate Calculations 

Hawaii's current four-year graduation rate methodology does not include students who 
transfer in after the cohort of first-time ninth graders is established. The state is now in 
the process of changing this procedure to include transfer-in students in the four-year 
adjusted cohort graduation rate calculation. 

The draft and final publishing of the October 2008 Title I regulations governing NCLB 
data and reporting triggered state action on this issue. 

As required by regulation, Hawaii plans to report the new adjusted cohort rate in its 
2011 State and School Accountability Reports (i.e., report cards). The 2007 ninth grade 
cohort that graduates in 2010 will be the lagged cohort reported in these reports. The 
new adjusted cohort rate will not be used for 2011 AYP determinations as allowed by 
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the regulations; at the time of this writing, Hawaii intends to use the current "old" 
graduation rate on the 2010 cohort for AYP determinations. 

At present the primary stakeholders impacted by this transition are those state 
educational agency offices charged with generating graduation rate calculations. These 
offices will be responsible for defining transfer-in and identifying the related impact on 
student registration procedures at the local level as data are entered into the state's 
information management system. Hawaii has decided to continue its original cohort 
graduation rate calculation that 
does not include transfers-in as 
well as initiate the calculation of 
the new adjusted cohort 
graduation rate that includes 
transfers-in. The intent of this 
dual set of calculations is to 
better identify the four-year 
impact a school has on those 
students experiencing the 
school's entire instructional 
program. However, Hawaii will 
continue to communicate the 
importance of helping all 
students reach proficiency, 
regardless of when they transfer 
into state schools. 

Training, professional 
development, and support will 
commence once 
recommendations are approved 
by the state superintendent and 
related documentation is 
prepared for peer review. 

At the time of this writing, 

Hawaii is reexamining the Annual Measurable Objectives (AMOs) established in 2002. 
Possible revisions include the incorporation of the extended adjusted graduation rate. 

Hawaii plans to monitor the use of the new adjusted cohort rate and concurrently 
compute the old graduation rate to analyze the differences. The use of a five-year 
extended adjusted cohort graduation rate will also be evaluated in the first two of years 
of implementation. 

Iowa's Decision to Expand the Collection of Enrollment Data 

To improve its statewide data system, Iowa expanded the enrollment data it collects, 
with a particular emphasis on the collection of information and data at the point when 



Hawaii established an adjusted graduation 
cohort workgroup composed of 
representatives from schools, state 
officials, experts in curriculum and 
instruction, and operational support 
specialists 

The workgroup reviewed the new graduation 
rate requirements, discussed options and 
issues, and proposed guideline 
recommendations to the state superintendent 
to facilitate the planning, development, and 
implementation of the adjusted cohort 
graduation rate. Should these 
recommendations be approved, the 
procedures, graduation rate targets and 
criteria will be submitted to ED for peer 
review in early 2010. Upon approval by ED, 
information about the new graduation 
methodology will be shared with 
stakeholders, including high schools, relevant 
department administrative offices, and the 
state board of education. 
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students enter and exit the public education system. In addition to requiring an exit 
code and date, destination codes and destination locations are also required. As a 
result, between-district transfers can be verified by matching exits with entries, and 
more accurate accountability decisions are rendered. 

The transition arose as a result of Iowa's decision to calculate and report more accurate 
cohort graduation rates. As a result of this transition, graduation rates may actually be 
reduced in some instances; however, Iowa recognizes the importance of basing policy 
and the related accountability 
decisions upon meaningful and 
reliable data. As such, the 
transition and the elated impact 
on local school districts were 
discussed beforehand with the 
attorney for the state 
department of education, 
district administrators, and other 
stakeholders charged with data 
collection and reporting. Iowa 
likewise found it useful to learn 
what has worked in other states. 

Iowa maintains a policy of 
communicating any and all data 
decisions as early as possible in 
order to familiarize all key parties with upcoming changes. Beginning in January, 
decisions regarding changes to current data elements as well as the addition of new 
data elements are discussed within the department of education. Phone calls with all 
student information system vendors are held during the month of March to discuss the 
next year's reporting requirement changes to the required extracts. Ongoing 
communications between the state and student information system vendors help 
ensure a timely and accurate release of the next year's reporting module. Training 
sessions with districts are held during April and completed by early May. The sessions 
involve communicating reporting requirements for the end-of-year submissions as well 
as previewing data reporting changes to be implemented in the next year. In August and 
September, training sessions are held to communicate changes to districts for the new 
school year. 

Training sessions vary in format. The state's fiber optic network, the Iowa 
Communications Network, has been utilized to provide statewide training to many in a 
short period of time with little travel required. Regional sessions have also proven to be 
popular, allowing face-to-face interaction. Approximately once every two or three years, 
a statewide conference is held on multiple data topics. The target audience includes 
district and building administrators, secretaries, technology directors, guidance 
counselors, and food service directors. 



Iowa utilizes a three-step data collection 
process: training, testing, and production 

At the same time training is occurring, the 
state's data collection site is available to the 
field to test their data. One month prior to the 
beginning of the real collection, the field may 
upload and process test files. This process 
allows them to identify data errors and 
incongruent data through a variety of available 
reports. Throughout the testing and final 
submission periods, staff members are available 
to provide assistance. All training materials and 
timelines for key dates are posted on the state's 
website. 
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Iowa has implemented a series of data validity checks. Validity checks are run at the 
student level at the time of data submission and at the district level before a district is 
allowed to sign off on the accuracy of the data. At the state level, Iowa is now 
implementing cross-submission validity checks at the conclusion of a submission period. 
Students reported as actively enrolled during one submission are reported as missing 
during the subsequent reporting period if no records were received from the same 
district. Students marked as graduates in the spring for whom records are reported in 
the subsequent fall collection are identified and resolved. The state also maintains a 
policy of documenting all data changes in case of system audits. 

Kansas' Implementation of an Opportunity to Learn Assessment 

Like those in many states, 

Kansas' high schools differ in 
their curricula and course 
sequences. A topic or subject 
covered in one grade in one high 
school, for example, may be 
addressed at a different grade in 
another. Yet prior to the 2006- 
07 school year, the state 
assessment for high school 
mathematics was given in grade 
10, and the reading test was 
given in grade 11. Kansas' 
implementation of Opportunity 
to Learn (OTL) gives local 
educators the flexibility to 
schedule state assessments in 
these subjects after students have had the opportunity to learn the knowledge, skills, 
and concepts addressed in Kansas' content standards. Moreover, OTL gives a second 
opportunity for students scoring below "meets standard." 

OTL was proposed by a former state commissioner of education prior to the advent of 
NCLB. Plans for high school history and science tests had also included a two-part test- 
students would be given partial tests in life science and physical science— each after 
completing the respective courses. Similarly, partial tests would cover U.S. History and 
World History. The parts, though administered on separate occasions, would be treated 
as the same test. 

Although Kansas had developed OTL beginning in the 2005-06 school year, the 
implementation of new state assessments that year postponed its launch until 2006-07. 
This also coincided with ED's decision to allow high school students who failed to meet 
state standards to be tested again. 

A great deal of communication and clarification regarding policy, guidance, and tracking 
of individual students has been an important and ongoing effort by the state education 



While it benefits Kansas' high school 
students and improves the accountability 
system, Opportunity to Learn (OTL) adds a 
new layer of complexity to the system 

New infrastructure had to be planned and 
built; scores and assessment results had to be 
stored or banked and new reports generated 
so that schools, districts, and the state could 
track which students had yet to be tested, 
which had failed to meet standards and were 
eligible for testing, which had completed the 
test, and which had not been tested. The 
agency's new rules had to cover all of these 
situations. 
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agency. For example, when a student is tested in school A and does not meet standard, 
and then transfers to school B, does school B have one or two opportunities to assess 
the student? If the student does not meet standard in school B, in which school— A or 
B— will the student's results count for AYP? If a student is officially enrolled in grade 10, 
and then, because of a large number of credits being awarded, officially becomes a 
grade 12 student, in what testing cohort is the student's results included? Can a student 
be tested more than once in a semester? 

Kansas' lessons that can be shared with other states undergoing or considering similar 
transitions include: 

• Anticipate all possible scenarios: While it benefits Kansas' high school students 
and improves the accountability system, OTL adds a new layer of complexity to 
the system. A clear set of business rules needed to be developed to cover them. 

• Communication: Establish clear channels of dissemination of the new rules and 
regular communication with schools and districts about any questions that arise. 

. Reporting: Generate reports that make it easy for schools to know the testing 
status of their students. 

• If possible, keep the rules consistent for each subject: Kansas' schools can retest 
a student who has failed to meet standards in mathematics or reading; however, 
they cannot do so in science, history and government, or writing. With two-part 
assessments in science and history and government, it is not practical, or valid, 
to make the claim that a student who failed a partial test on the first opportunity 
should be retested on a partial test. Variations in rules by subject can be a source 
of confusion. 

Kentucky's Development of a New System of Standards and 
Assessments 

In early 2009 the Kentucky State Legislature passed Senate Bill 1, requiring the SEA to 
rewrite its content standards and develop new assessments for state and federal 
accountability. The state accountability system— which had been in effect since 2000 
and set biennial targets for schools through the 2013-14 school year— was eliminated. 

The previous system included assessments in seven content areas (reading, 
mathematics, science, social studies, writing on-demand, arts/humanities, and practical 
living/vocational studies) with an additional writing portfolio assessment. The results 
from these assessments along with results from PLAN, ACT, and nonacademic indicators 
(attendance, dropout, retention, transition, and graduation rates) were included in the 
state accountability index. 

Bills introduced in prior legislative sessions proposed substantive changes to assessment 
and accountability systems. Senate Bill 1 appeared to be the result of growing 
agreement that the assessment system was taking up too much of the instructional time 
available to students and teachers; additionally, there were longstanding concerns 
about the state's locally assessed writing portfolios. 
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Senate Bill 1 replaced the arts/humanities, practical living/vocational studies, and 
writing portfolio assessments with a program review system to ensure schools 
continued delivering instruction 
in those subjects. Until a new 
state assessment system is 
created in 2012, Kentucky will 
rely on the accountability 
provisions contained in NCLB for 
all schools and districts, whether 
or not they receive federal Title I 
funds. Kentucky underwent an 
earlier transition with the 
redesign of the state assessment 
system as a result of the federal 
requirement for annual reading 
and mathematics assessments in 
grades 3-8 and in high school. 

Through 2005-06, the final year 
of Kentucky's contract with its 
existing assessment vendor, the 
state had used an augmented norm-referenced test (NRT) to meet NCLB requirements. 
In 2006-07 Kentucky transitioned to a new testing vendor, new standards, and a new 
test design. 

In the 2006-07 and 2007-08 school years, Kentucky implemented a concordance model 
approach in order to maintain historical trend data over a multiyear period as requested 
by the Kentucky Board of Education. However, the SEA had difficulty communicating the 
analysis and the use of the concordance approach with educators and the public. As a 
consequence, beginning with the 2008-09 school year the board revised baselines and 
established targets for state accountability purposes using the prior two years from the 
new assessment. 

Throughout this transition, Kentucky has learned these lessons: 

• Shaping policy is as important as policy implementation: During Kentucky's 
transition, a key facet of Senate Bill 1 discussions was the role of SEA staff in 
providing input, which helped to shape the final bill. SEAs should work with their 
legislatures to reach a compromise with stakeholders (e.g., superintendents and 
state legislators) as soon as changes are proposed. 

• Monitor the implementation of the accountability system: If aspects of the 
accountability system appear to lack stakeholder support, address those issues 
proactively. For example, given concerns about Kentucky's locally assessed 
writing portfolios, the SEA may decide to make changes amendable to 
stakeholders yet preserve the integrity of the system (e.g., address time out of 



Communication 

Throughout its transition, Kentucky has 

learned 

• shaping policy is as important as policy 
implementation 

• monitor the implementation of the 
accountability system 

• when possible, take the long view 

• use varied means of communication 

• in linking different assessments for 
accountability purposes, avoid 
communicating changes in highly 
statistical terms 
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instruction and teacher training issues, consider analytical versus holistic 
approaches to scoring, consider standard setting process, etc.). 

• When possible, take the long view: When responding to a change in the 
accountability and assessment system, consider the implications of those 
changes in future years with respect to communicating information about the 
changes to local school districts in timely ways, and in gauging the workload (at 
the state and local levels) from development through implementation of the 
new system. 

. Use varied means of communication: Kentucky has communicated changes to 
the accountability and assessment system via regular mail, online WebEx 
meetings, and regional face-to-face meetings with assessment coordinators at 
the local level. 

• In linking different assessments for accountability purposes, avoid 
communicating changes in highly statistical terms: Although technical advisors 
may agree on the technical quality of such an approach, the public perception 
and interpretation of the approach should also be considered. 

Massachusetts' Transition from a Competency Determination Rate 
to a Four-Year Graduation Rate as the Additional AYP Indicator 
for High Schools 

Beginning with the 2007 AYP determinations, Massachusetts transitioned from using the 
grade 12 competency determination rate (the percent of students eligible to graduate 
as of their senior year) to a four-year cohort graduation rate as the additional AYP 
indicator for high schools. 

Massachusetts applies the graduation rate standard to every student group that meets 
minimum reporting size requirements. To make AYP in 2007 and beyond, a high school 
group is required to meet the 95 percent participation requirement, either the state's 
performance requirement or safe harbor, and the state's minimum graduation rate 
standard for the given year. 

Massachusetts developed a student information management system beginning with 
the 2002 school year, and one goal was to establish an on-time graduation rate as soon 
as possible. The goal was complicated further by state desire to label five-year 
graduates as on time. 

Massachusetts began calculating and reporting cohort graduation rates in 2006 as part 
of overall efforts to improve educational outcomes for all students and to use the 
cohort rate for federal AYP determinations. Massachusetts, along with other states, had 
committed to utilizing four-year cohort graduation rate data according to the 
methodology outlined in the National Governors Association's Graduation Counts 
Compact on State High School Graduation Data. 

Until 2006, graduation rates for Massachusetts high schools could only be estimated 
from annual dropout data or from grade-level enrollment information. By 2006, 
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however, the state had collected a sufficient quantity of longitudinal student-level data 
via its Student Information Management System (SIMS) to be able to track individual 
students from their initial entrance into ninth grade through graduation. 

At its February 2007 meeting, the Massachusetts Board of Elementary and Secondary 
Education voted to establish a minimum four-year graduation rate standard of 55 
percent as the "must meet" AYP target for all public high schools. The 55 percent 
standard, used in 2007 AYP determinations, was applied to data from the 2006 
graduating cohort. 

In 2008 the board voted to raise the four-year standard to 60 percent and to apply that 
standard to the 2007 graduating cohort. Student groups that did not meet the 60 
percent standard could also make AYP by showing an improvement of at least two 
percent between 2006 and 2007. These criteria applied to 2008 AYP determinations. 

This transition affected the second indicator for high schools. When the initial set of 
graduation rate data was released to the public in February 2007, Massachusetts found 
that in 209 of the state's 279 school districts with high schools, at least 80 percent of 
students in the class of 2006 graduated within four years. And in 104 districts more than 
90 percent graduated within four years; in 35 districts more than 95 percent graduated 
within four years. Despite this positive news, only 62.3 percent of students in urban 
communities statewide graduated within four years. The districts with the lowest 
graduation rates included Lawrence (41 percent), Chelsea (45.8 percent), Holyoke (49.4 
percent), Springfield (51.2 percent). Fall River (54.2 percent). New Bedford (57.4 
percent), and Boston (59.1 percent). 

Given the differences in performance among Massachusetts' communities, coupled with 
the state's commitment to include all student groups in AYP determinations for this 
indicator— a policy not required under NCLB— the Massachusetts Department of 
Elementary and Secondary Education's accountability and targeted assistance group 
worked closely with the data collection group to recommend a reasonable four-year 
standard to the board for approval. 

As 2007 was the first year of implementation, and calculating improvement from the 
previous year was not possible, the minimum graduation rate target was set at 55 
percent, which was comparable to the previous target using the competency 
determination rate. In its July 2007 and August 2008 decision letters to Massachusetts, 
ED approved the state's 2007 and 2008 AYP targets with the expectation that 
Massachusetts set a more challenging graduation rate target in future years. 

Massachusetts can share the following insights from its transition: 

• A major challenge was defining what is meant by on-time graduation and its 
relationship to "the standard number of years" described in Section 
llll(b)(2)(C)(vi) of NCLB. The state explored basing such judgments on 
individual expectations regarding the expected time it will take each student to 
graduate, but concluded that this approach was not appropriate because it can 
lead to lower expectations for students, be difficult to implement, and create a 
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lack of transparency and comparability in the final data. Therefore, the SEA 
decided to publish a straightforward four-year graduation rate in 2006, a five- 
year graduation rate in 2007 and beyond, and additional rates as policy and 
program needs may warrant. Rates are generated for the entire student 
population and for individual student subgroups at the state, district, and school 
levels. 

• The SEA recognized the 
need to report data in a 
timely way, but that 
objective had to be 
balanced with ensuring 
that the reported rates 
are as accurate as 
possible. This was 
particularly true in 2006, 
when the data were first 
used for high-stakes 
findings. The SEA began 
collecting student-level 
data through SIMS for 
longitudinal analysis in 
the 2002-03 school year. 

The 2006 cohort 
graduation rate 
calculations included 
data going back to the 
inception of SIMS, when 
districts were still becoming familiar with the system. The SEA had no way to 
know whether the students in the first SIMS data collection were first-time ninth 
graders. The rates would have fluctuated substantially between 2006 and 2007 
because large percentages of students are retained in ninth grade in 
Massachusetts. Consequently, the SEA allowed for the possibility of a limited 
number of corrections. Student-level data making up the 2006 graduation rate 
were released to districts in the fall of 2006, and district staff had approximately 
one month to review and request corrections to the data. These data were 
provided to districts via the Security Portal— the SEA's secure, online data 
transmittal application used by authorized school and district personnel to 
submit and review data. The SEA reviewed all requests and identified limited 
instances in which changes to the data were warranted to ensure accuracy. 

• Massachusetts wanted the completed diploma to clearly represent that a 
student had met local and state standards, whether it took four years or more 
to meet those standards. Relying on an AYP indicator that valued only four-year 
graduation rates contradicted that state policy, but little flexibility was initially 



While the key goals of Massachusetts' 
accountability system have not changed 
due to this transition, the transition has 
highlighted the need to ensure that 
accountability is reciprocal: for every unit 
of accountability demanded of school and 
district leaders, the state should strive to 
provide a corresponding set of supports 
and interventions 

Examples include convening stakeholders 
from across the state to address graduation 
rates with respect to AYP determinations and 
to build school/district capacity in increasing 
graduation rates and piloting an early 
warning system to help local educators 
identify and intervene with students at risk of 
not graduating on time. 
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offered by ED on this matter. Massachusetts also had a strong desire to calculate 
the on-time graduation rates by subgroup, even though ED did not require it. 

• Public reporting of results can increase stakeholder understanding of and 

involvement in helping students graduate from high school. The new rules were 
described in numerous memoranda and conference calls as well as integrated 
into trainings at the local level. In addition, the state's four-year graduation 
report attracted a great deal of press; the report helped describe a problem that 
had been masked by the relatively low annual dropout rates. The public had not 
yet grasped the cumulative effect of dropouts and retentions. 

At its February 2007 meeting, the board voted to establish a Graduation Rate Taskforce 
comprising representatives from business and industry, school districts, high schools, 
alternative education programs, teacher organizations, student organizations, private 
non-profits, and SEA staff. The taskforce met three times over the course of six months 
to review additional data related to the high school graduation rate and to consider 
other issues, such as making recommendations for AYP improvement targets and 
addressing capacity and resources needed to increase the percentage of students 
graduating from high school. The taskforce collected research to identify the reasons 
students drop out of school; identified what steps Massachusetts could take to increase 
college and career readiness, as well as to increase graduation rates; and developed 
recommendations on policies and programs that could make a positive change in high 
school graduation rates. The taskforce identified a primary need to increase the number 
of high-quality pathways for students who are most at risk of not graduating, and for 
bringing back students who have dropped out of school. 

AYP reports for a given year show graduation rates for the previous year's cohort; for 
example, 2007 AYP reports showed graduation rates for the 2006 cohort. While using 
data from for the previous year's graduating cohort allowed the SEA to use a data set 
for high-stakes purposes that had been thoroughly reviewed by district and SEA staff, 
these graduation rates alone are of limited utility to stakeholders because they are 
"lagged" indicators— the population measures the educational outcomes of students 
who already graduated or dropped out of school by the time the data are reported. In 
spring 2008 the SEA piloted an early warning system for the state's 24 urban districts. 
Called the Early Warning Indicator Index, the system is intended to help local educators 
identify high school students at risk of not graduating on time so that proactive 
measures can be taken to make timely interventions in educational programming for 
these students. In addition to identifying individual students for intervention, the index 
uses a set of core indicators based on data from all districts— therefore applicable across 
all schools and districts— and provides the data in a user-friendly format for 
presentation and analysis at the local level. 

The index has appeal because it assists schools and districts with issues over which they 
have some control, such as aspects of their organizational and programmatic design. 

The index remains a work in progress as the SEA investigates additional statistical 
techniques to improve the validity and reliability of the system. 
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The SEA continues to investigate other options for the additional improvement indicator 
for the AYP graduation rate, with an emphasis on factors local educators may be able to 
address in timely ways. These options include showing an increase in the grade nine 
attendance rate from one year to the next (some studies have demonstrated a 
relationship between freshman year attendance and on-time graduation) and showing a 
reduction in the high school dropout rate from one year to the next. The SEA will 
continue to explore the benefits and limitations of these possible approaches in the 
coming months and years. 

Michigan's Development of a Work Skills Assessment and College 
Entrance Examinations 

Michigan legislation passed in 
2005 required a work skills 
assessment and a college entrance 
examination as components of the 
high school assessment. The 
legislation also required 
compliance for approval of the use 
of the high school test under 
NCLB. This meant, in essence, that 
augmentation would be required 
to round out alignment of the new 
test to Michigan's high school 
content standards. 

The Michigan Association of 
Secondary School Principals 
(MASSP) has partnered for many 
years with ACT, Inc., and high 
schools are approved by ACT as 
test centers on Saturdays. ACT 
reported that approximately 70 percent of students in Michigan took the ACT. MASSP 
stated that students applying to college were motivated to do well on the ACT, while 
students were not motivated to do well on the state high school assessment. In 
addition, MASSP claimed that making the transition would save the state significant 
funds— it would be less expensive to administer the new assessment than to administer 
the old assessment. 

The state department of education initially opposed the proposal for several reasons: 

. Despite the MASSP claim that the transition would save money, the SEA 
projected a manifold increase in overall costs based on cost estimates for the 
multiple components. 

. The increased strictness of the administration procedures would cause more 
schools to have invalid scores, leading to more schools not making AYP. 



Advocacy by the High School Principals 
Association was the primary force 
underlying Michigan's adoption of a work 
skills assessment and college entrance 
examinations 

Michigan has historically been an ACT state 
for college admissions. All of the major 
universities, including private colleges, use 
the ACT for admissions. ACT had also 
previously worked with the state of Illinois to 
develop a state high school assessment that is 
structured to include the ACT, WorkKeys, and 
a state component. The Michigan Association 
of Secondary School Principals (MASSP) and 
ACT devoted considerable resources to 
advocacy for this proposal. 
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. The reduced flexibility in scheduling and carrying out assessment activities would 
place a significant new burden on schools. 

. The requirement for augmentation to provide for adequate alignment to 

Michigan's high school content standards would result in a longer test than was 
previously administered. 

However, the governor's office supported the proposal because it would provide 
baseline information and a measure of progress on efforts to increase the percentage of 
Michigan high school graduates that are prepared for postsecondary success. 

The legislation was passed in 2005, a pilot was carried out in spring 2006, and the full 
transition occurred for the spring 2007 assessment. 

Because ACT won the competitive bid, the fully customized state high school 
assessment was replaced by a regimented three day testing process, with day one being 
composed of the ACT + Writing test, day two being composed of WorkKeys 
assessments, and day three being composed of Michigan-specific augmentation to 
round out alignment to Michigan's high school content standards. This test was named 
the Michigan Merit Examination (MME). 

Schools had to transition from a three-week testing window to giving the test to all 
students on the same day for each of the three days. Students who missed the test days 
are allowed to take a makeup for each missed day exactly two weeks later. Schools also 
had to transition to the increased rigor of becoming established as an ACT test center, 
including severe consequences for mis-administrations. Schools also had to transition 
from appealing to the SEA on issues of invalidated scores to appealing to ACT. 

The change in the assessment required analysis to determine whether the AYP annual 
measurable objectives (AMOs) needed to be reset. From the results of the standard 
setting activity, the state board of education adopted proficiency cut scores that were 
approximately equivalent in rigor to the cut scores from the previous, fully customized 
high school assessment. Therefore, the AMOs were not reset, as the impact of the new 
assessment on AYP calculations was minimal. The transition did have an impact in other 
areas, as noted below: 

. The transition had significant cost implications. The new high school 

assessment costs were significantly higher than the costs of the previous fully 
customized assessment. The previous assessment cost the state $19 per student. 
The ACT component of the new MME alone costs the state $47 per student; the 
WorkKeys component costs $15 per student; the augmentation costs 
approximately $5 per student; and the project management, IT requirements, 
psychometrics, and reporting systems necessary to create a single score for each 
subject using all components of the test cost $58 per student. 

• The transition had significant implications for individual students and schools. 

In the first years of the program, many scores were invalidated on an individual 
or schoolwide basis because of prohibited behavior or mis-administration. While 



Key Elements for Educational Accountability Models in Transition 



Page 22 of 31 




the SEA would have made the same decision as ACT, in many cases, significant 
numbers would not have been deemed invalid by the state under previous 
policies. Therefore, some students did not receive valid scores, and some schools 
did not make AYP because of the new, stricter policies on test administration 
that came with using ACT products. 

• The transition had an impact on the availability of retesting opportunities for 
high school students. With the previous test, students could retest in the fall or 
spring of the next year. The MME initially allowed for retesting in fall or spring. 
However, the fall retest period was eliminated because of prohibitive costs for 
an additional cycle involving the ACT products and the untenable burden on 
schools of two test cycles per year with the new strict requirements. The spring 
retest was also largely eliminated through legislation because of the prohibitive 
costs. 

• The transition had an impact on students with disabilities (SWDs) and English 
language learners (ELLs). Whereas states must comply with NCLB, the 
Individuals with Disabilities Education Act (IDEA), and the Americans with 
Disabilities Act (ADA), ACT's policy has been to comply only with ADA. This 
resulted in many of the state-offered accommodations not being allowed by ACT 
if students desired to get an official ACT or WorkKeys score report. When an 
Individualized Education Program (IEP) designated an accommodation allowed 
by the SEA, but not by ACT, the student received official MME scores, but no 
official scores on the ACT or WorkKeys. ACT does not approve accommodations 
specifically for ELLs. Therefore, all accommodations provided because of ELL 
issues resulted in official scores for MME, but no official scores for ACT and 
WorkKeys. 

• The transition had effects on the usefulness of the data, because individual 
item data could not be provided to schools. Because ACT products are 
proprietary, no item information could be provided to schools. Many schools 
have lamented the loss of the individual item data. 

Ultimately, however, the transition was successful. Part of the reason for the success 
was a significant ongoing communications and training campaign to keep all 
stakeholders apprised of the progress of the transition, administration procedures, 
registration procedures, and of the new requirements that would become a part of the 
new test. 

Throughout this transition, Michigan learned the following lessons: 

• Make sure all schools are identified and trained as ACT centers. 

• Train heavily on accommodations and timing codes. 

• Formally include vendor compliance with all state and federally required 
legislation applicable to the SEA. 
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• Carefully examine and evaluate claims on cost and impacts on schools and 
students. 

Numerous formal communications to the field, to district administrators and to high 
school principals have formed the communications strategy. Training has been provided 



on test administration, student registration, and data use. 

Minnesota's Implementation of a "Second Generation" of High 
School Assessments 


Minnesota is in the process of 
implementing a "second 
generation" of assessments as 
part of the high school 
graduation requirements. The 
state legislature first required 
graduation tests for Minnesota 
students in 1996. In 2003, the 
state Academic Standards were 
revised and a new generation of 
graduation tests was required. A 
new writing test was 
implemented in 2007, a new 
reading test in 2008, and a new 
mathematics test in 2009. Also, 


Since the change occurred within the 
legislative session, numerous stakeholders 
were aware of the change: it was a closely 
watched legislative discussion 

Subsequent to the bill passing, the SEA 
provided documentation about the change in 
legislation: 

httD://www.education.state.mn.us/mdeDrod 




/srouns /Assessment/documents /FAO /0 146 




39.pdf. 

The SEA also developed significant 
documentation about the new high school 
assessment system: 

httD://www.education.state.mn.us/MDE/Acc 




the first administration was 


ountabilitv Programs /Assessment and Testi 




changed from grade 8 for 
reading and mathematics and 
grade 10 for writing to grade 9 


ng/Assessments / GRAD / index.html. 







for writing, grade 10 for reading, 
and grade 11 for mathematics. 

The law (both statute and rule) required several changes: 

• Rules for special education students were restricted. 

• Rules for new-to-country English language learners were modified. 

• Rules for new-to-state students via an assessment taken in a prior state were 
implemented. 

Standard setting for reading in 2008 established an expectation that the passing rate for 
the graduation tests was equivalent to the proficient level on the NCLB Title I 
assessment originally set in 2006. This has caused significant concern to be raised in 
anticipation of this spring's implementation of the mathematics assessment, because 
the 2008 proficiency level for high school mathematics was about 34 percent. Also of 
significant concern is the later time for first administering the reading and mathematics 
assessments in a student's high school career— less time is available for remediation. 
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While the first graduating class required to pass these assessments will graduate in the 
spring of 2010, changes have already been enacted. Due to concern over the excessive 
failure rate expected of the mathematics test, the legislature passed a five-year 
moratorium that no longer required students to pass the mathematics test to graduate. 
(Students still have to pass the reading and writing tests.) In its 2009 session, the 
Minnesota legislature created a work group to study the effects of high-stakes 
graduation tests in Minnesota and future directions for these requirements. A critical 
lesson here is that a broader group of stakeholders involved in the conversation over 
time is essential. Previous changes occurred without a sufficient number of key 
stakeholders aware of the potential consequences of moving the graduation 
requirement to the high school level and with a higher expectation of proficiency for 
graduation. 

The SEA is developing enhancements to its statewide data warehouse for collection of 
the alternate pathway in mathematics scheduled to be available in early 2010. Districts 
will enter this coding for students graduating under the alternate pathway. The SEA will 
validate that the student has attempted the assessment at least three times as required 
in legislation. The remaining two requirements are the responsibility of the school 
district and are subject to audit at the discretion of the SEA. 

Nebraska's Transition from a System of Local Assessments to Single 
Common Tests in Core Academic Subjects 

State legislation was introduced in 2007 and 2008 that required standards revision and 
state assessment development. The legislation called for single common assessments in 
reading, mathematics, and science to begin in the 2009-10 school year, with each 
subject area to be phased in over time. The writing test remained in the law, but the use 
of local assessment data for accountability reporting was eliminated. Basically the law 
was a mirror of NCLB, requiring annual testing in grades 3-8 and in high school. 

Nebraska signed a compliance agreement with ED allowing the state to receive NCLB 
funds so long as documentation would be provided that the new tests are being 
developed and implemented according to the timeline specified by the Nebraska 
legislation. The test results are to be reported by score and subscore. 
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The state worked for many 
years to obtain federal 
approval for using the 
established local 
assessments in calculating 
AYP. One goal was to 
maintain as much district 
control of assessments as 
possible. As the new system 
is being developed, 
compliance with NCLB is a 
primary goal. Nebraska 
anticipates increasing the 
state's role in publishing 
school and district 
accountability information as 
the new state assessments 
become operational. 

In 2000 Nebraska built a 
locally based assessment 
system, the School-based Teacher-led Assessment and Reporting System (STARS). Under 
the STARS approach, local school districts developed classroom-based assessment to 
measure student achievement on state-developed standards in reading, mathematics, 
science, and social studies. In addition to reporting student results on the standards, 
districts submitted their local assessments to the state for review and for a public rating. 
Local assessment was only one part of the assessment system, as districts were required 
to participate in the statewide writing test and to administer a norm-referenced test in 
at least three grades in their districts. The approach allowed the state to gather multiple 
indicators: student achievement on standards, statewide writing results, assessment 
ratings, and norm-referenced test results. These multiple indicators were used to make 
comprehensive decisions about the quality of the schooling and accountability, and 
those decisions were based in the accreditation rules. 

Although the system was effective, comprehensive, and balanced, there were 
downsides to the combination of the local and state approach. The system did not allow 
direct comparison between school districts on the same common measures. The 
system, said some, was too complex, involving too many data factors. It was not simple 
with a single "bottom line" state test. 

Since the primary purpose of the system was that of improving student achievement, 
not comparative accountability, it became clear that the bottom line of accountability 
was direct comparison between school districts, and that could only be achieved by 
single common measures. Coupled with the fact that Nebraska remained one of the few 
states that had not achieved federal approval under NCLB with its STARS system, the 
political winds began to shift in the state. 



In short, Nebraska is changing everything: 
policy, practice, and politics. With the 
change in assessment comes the change in 
accountability 

With the change in accountability comes a 
change in culture. Districts and the department 
of education are caught between two systems: 
the old and the new, with a gulf of transition 
between. Each year when a new test is phased 
in, a piece of the old system goes away. The 
years of transition will be continuous through 
2012. Both the state and local school districts 
are working on extremely short timelines. 
Complicating the situation is that the 
assessment and accountability transition 
occurred at the same time transitions were 
being made in senior leadership at the state 
level. 
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Costs are always an issue, but the legislation also brought additional dollars to 
assessment. In addition, two-thirds of the new state system will be funded with federal 
funds. 

Nebraska can lend the following insights from its transition: 

• The state will persevere. A new commissioner took over in the spring of 2009. 
The department is providing the message "Keeping the Focus, Expanding the 
Vision, Finding the Balance." Information as it is known is shared with the field 
purposefully and completely. The steps of test building are underway and on 
track. 

. The state is using its finest resource, assessment-literate educators, as the 
backbone of standards revision and test development. Because of the 
knowledge and expertise teachers acquired through the STARS process, they are 
instrumental in the design and reworking of test items. 

• A contractor has been secured as a competent partner, and advice is sought 
from external experts. Although everything is changing, SEA leadership is now 
stable, reassuring local school districts and encouraging them to maintain local 
assessments for use instructionally. 

. The state has an extensive communication plan. Nebraska used telecasts, video 
streaming, paper documents, speaking engagements, web postings, and its 
service unit network. Nebraska is also planning professional development 
throughout the upcoming year as well as professional development within 
regional service units. 

. The legislation requires verification studies, and the state has a long-standing 
evaluation contract with the University of Nebraska as well as many external 
experts. 

The state board of education is undergoing monthly discussions about a new policy 
framework for standards, assessment, and accountability. Meanwhile, the state 
department of education maintains a focus on student learning. 
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West Virginia's Development of New Standards and Assessments to 
Assess 21 st -Century Skills 



The West Virginia Department of 
Education transformed the 
state's educational system into a 
high-quality global system of 
education that is current, 
engaging, relevant, and exciting 
to 21 st -century students. The 
goal of this transformed system 
was to develop in West Virginia 
students more complex 
communication skills, increased 
critical thinking and problem 
solving abilities, greater decision 
making skills, and the ability to 
thrive in a globally competitive 
21 st -century world. The 
development of West Virginia's 
21 st -century learning plan— 

Global 21 — provided the 
framework for this systemic 
approach to helping children 
learn by providing rigorous 
instruction presented at a 
variety of depths of knowledge 
(DOKs), integration of 
technology tools, and balanced 
assessments that would facilitate and invigorate student inquiry and learning. 

To meet the goals of Global 21, West Virginia began the lengthy and vigorous process of 
developing new state content standards and objectives (CSOs) to include increased rigor 
and a variety of DOK levels. To measure student achievement of these content 
standards, the SEA Office of Assessment, Accountability, and Research undertook the 
goal of developing a new statewide accountability assessment, WESTEST 2, which would 
align to the new state CSOs and would more accurately measure student achievement 
in grades 3-11 in reading/language arts, mathematics, social studies, and science. The 
first operational WESTEST 2 was administered in the 2008-09 school year. 

Prior to 2008, a review of West Virginia's content standards by national experts revealed 
that the state's CSOs lacked the rigor necessary to meet the challenges of the National 
Assessment of Educational Progress, the Trends in International Mathematics and 
Science Study, and other national and international assessments. This finding was not 
acceptable for a state that desired its students to be globally competitive and lifelong 



The West Virginia Department of 
Education worked diligently with local 
school districts to provide a 21 st -century 
systemwide approach to assessment 
within the state 

The SEA funded and provided a network of 
high-quality support tools, including 
techSteps, Acuity, Writing Roadmap 2.0, 
INTEL, Thinkfinity, Teacher Leadership 
Institutes, Special Education Teachers 
Leadership Academies, Principal Leadership 
Institutes, county team conferences, and other 
supports to assist local school districts in 
implementing the goals of Global 21. Further, 
the SEA recognized that teachers, principals, 
and other leaders required high-quality, 
sustained professional development that 
involved emerging strategies and knowledge 
in areas such as instruction, technology, and 
assessments. To that end, the SEA has worked 
to provide an array of these types of 
professional development opportunities for 
educators. 
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learners. By early 2005, the state was poised for major changes within its accountability 
system. 

In 2005, West Virginia became the second state in the nation to implement the 
Partnership for 21 st -Century Skills (P21) model. P21 is an advocacy organization that 
includes members from the business community, education leaders, and policymakers. 
This implementation and support further strengthened West Virginia's efforts in 
developing new state content standards. 

In July 2008, the new CSOs became effective for use in every West Virginia classroom, 
and the revisions to these content standards and objectives significantly broadened the 
scope of the state's curriculum. More recent external reviews of state CSOs by local and 
national experts have identified our state curriculum as meeting world-class status. 

Many policies other than the content standards have been revised to align with the 
overall initiative, including accountability policies. The SEA developed a performance 
index for school accreditation that is consistent with 21 st -century schools, developed a 
new assessment system to measure the new content standards, and reorganized some 
divisions within the SEA to implement the changes in the curriculum. 

Because West Virginia made systemwide changes to include the development of new 
CSOs and a new aligned statewide accountability assessment, WESTEST 2, the state 
increased the overall rigor of these standards and assessments, which called for a 
resetting of performance standards (cut scores on WESTEST 2). To that end, in April 
2008, West Virginia proposed two amendments to make a substitute trajectory to AYP 
for 2009 and 2010 and requested approval for these changes from ED. This request was 
made approximately one month prior to the first administration of the new WESTEST 2 
in May 2008. In essence. West Virginia requested approval to use the same substitute 
for determining AYP for 2009 and 2010 school years as approved in the original 
Accountability Workbook and reset the trajectory in fall 2010. 

In August 2009, ED approved West Virginia's amended accountability plan and posted 
those changes on the SEA website. As a result, the state reset its starting points based 
on its 2009 assessments to establish new Annual Measurable Objectives (AMOs) using 
the statutory process laid out in ESEA. West Virginia would then reset its starting points 
using the results of the tests to be administered in 2009-10, average the 2009 and 2010 
starting points to establish AMOs for making AYP determinations in 2009-10, and create 
intermediate goals and AMOs that would result in all students meeting or exceeding the 
state's proficient level of achievement by the 2013-14 school year. 

West Virginia must submit the revised starting points set following the 2009-10 
administration of the assessments, intermediate goals, and AMOs to ED for review and 
approval before they can be used in making AYP determinations. 

West Virginia learned many lessons from engaging in the process of changing an entire 
accountability system, including: 
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• Such a process requires a shared vision by all educational stakeholders, 
tremendous work, sustained diligence in overseeing and completing the 
process, and a willingness to involve teachers, local and national experts, and 
other stakeholders in this reinvention process in order to make our state 
accountability system world class. 

• Systemwide change requires a great deal of political will and determination to 
move forward in increasing the rigor and depths of knowledge of content 
standards while realizing that school assessment performance will not likely 
measure up in the short term. Parents, business and community leaders, 
policymakers, teachers, and other educational leaders will seek answers to why 
students' scores are lower, and they will need instruction and support in 
understanding how the increase in academic rigor will ultimately benefit all of 
the students and citizens in West Virginia. 

• Professional development in clarifying and increasing understanding 
concerning increased rigor of standards is essential for all stakeholders, 
including schools, teachers, parents, and the general public. The SEA has 

worked with local school districts in providing a network of high-quality support 
tools and other supports to assist in implementing the goals of Global 21. 

• There is both a need and a challenge to make pertinent information available 
to all educational stakeholders. The SEA foresaw that it would need to educate 
state citizens about why there was a need for public education change within the 
state and how those changes might impact the performance data of schools. 
Thus, in 2009, the SEA undertook a major public relations campaign to get Global 
21 information out to the public. The SEA developed a website 
( http://wvde.state.wv.us/global/publications/ ) that provided one-page flyers 
containing basic information for teachers, parents of pre-K-4th graders, parents 
of 5th-8th graders, and parents of 9th-12th graders on topic areas including 
why is public education changing, how is public education changing, and how do 
we measure progress? In addition, the website contained a copy of the 
newspaper ad that ran in local newspapers and that provided pertinent 
information concerning Global 21 initiatives, as well as other tools including 
screensavers and wallpapers featuring Global 21. 

West Virginia's accountability policies were revised to incorporate the 21 st -Century 
Schools Partnership initiative. A performance index for accountability and accreditation 
(outside the NCLB model) has been developed and this index utilizes multiple 
performance measures using a compensatory model that is not dependent on only one 
subgroup or on one low score being the deciding factor. 

West Virginia is expecting student improvement across the continuum of learning rather 
than just moving students to mastery. For example, the newly developed index gives 
extra credit for getting students to above mastery and distinguished levels of 
performance. 
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Using data collected through the West Virginia Education Information System (WVEIS), a 
management information system that is online, interactive, and operates over a 
privately addressed intranet, the SEA collects from school files the information needed 
for state and federal reporting and decision making. From this collected data. West 
Virginia publishes a state report card as required by state law. An NCLB report card for 
schools and districts is published annually according to NCLB requirements for state 
reporting. 



Conclusion 

A decade ago, it would have been reasonable to expect that a paper on educational 
accountability models would emphasize goals, indicators, decision rules, and 
consequences as the primary components of accountability. After all, the fundamental 
premise underlying standards-based reform is that if you set high academic standards, 
design assessments to measure student progress toward them, and hold school and 
district leaders accountable for the results, then student performance will improve. 

States' experiences with designing and implementing accountability models since the 
inception of standards-based reform strongly suggest that communication and support 
are increasingly becoming the focal points of accountability, and that communication is 
particularly important when the accountability system is undergoing transition. 
Moreover, as educational accountability models mature and evolve, communication has 
increasingly been defined to include training and support in addition to reporting. 

Whether the focus is on designing better assessments, improving data collection 
procedures, or helping students become college and career ready, we anticipate a 
continued shift from emphasizing consequences and sanctions to the provision of 
appropriate supports to cultivate effectiveness. 
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